Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixing Mysqld.normalizedSchema() by formally parsing the query #13866

Closed
wants to merge 30 commits into from
Closed

Fixing Mysqld.normalizedSchema() by formally parsing the query #13866

wants to merge 30 commits into from

Conversation

shlomi-noach
Copy link
Contributor

Description

This fixes #13865. The existing query editing mechanism is a naive string search & replace, which can lead to incorrect replacement of table names. This PR will work to formally parse the query and only replace DB names.

This will be done in stages. First, we introduce unit testing to support the existing behavior. Then, introduce unit tests that break with #13865. Last, we will introduce formal parsing that passes all tests.

Related Issue(s)

Fixes #13865

Checklist

  • "Backport to:" labels have been added if this change should be back-ported
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on the CI
  • Documentation was added or is not required

Deployment Notes

@shlomi-noach
normalizedSchema: slight refactoring and adding unit tests
3c5501e 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach requested a review from dbussink August 28, 2023 10:23
@vitess-bot
Copy link
Contributor

vitess-bot bot commented Aug 28, 2023

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

  • Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

  • Apply the release notes (needs details) label if users need to know about this change.
  • New features should be documented.
  • There should be some code comments as to why things are implemented the way they are.
  • There should be a comment at the top of each new or modified test to explain what the test does.

New flags

  • Is this flag really necessary?
  • Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
  • vtctl command output order should be stable and awk-able.

@vitess-bot vitess-bot bot added NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels Aug 28, 2023
@shlomi-noach
introduce a test case that breaks
0acb7d9 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@github-actions github-actions bot added this to the v18.0.0 milestone Aug 28, 2023
@dbussink
Copy link
Contributor

There's a few other places where we use string replacements, but where really parsing should be used:

vitess/go/vt/mysqlctl/schema.go

Lines 90 to 102 in b65cde1

func (mysqld *Mysqld) GetSchema(ctx context.Context, dbName string, request *tabletmanagerdatapb.GetSchemaRequest) (*tabletmanagerdatapb.SchemaDefinition, error) {
sd := &tabletmanagerdatapb.SchemaDefinition{}
backtickDBName := sqlescape.EscapeID(dbName)
// get the database creation command
qr, fetchErr := mysqld.FetchSuperQuery(ctx, fmt.Sprintf("SHOW CREATE DATABASE IF NOT EXISTS %s", backtickDBName))
if fetchErr != nil {
return nil, fetchErr
}
if len(qr.Rows) == 0 {
return nil, fmt.Errorf("empty create database statement for %v", dbName)
}
sd.DatabaseSchema = strings.Replace(qr.Rows[0][1].ToString(), backtickDBName, "{{.DatabaseName}}", 1)

vitess/go/vt/mysqlctl/schema.go

Lines 438 to 445 in b65cde1

for _, td := range originalSchema.TableDefinitions {
if td.Type == tmutils.TableView {
// Views will have {{.DatabaseName}} in there, replace
// it with _vt_preflight
s := strings.Replace(td.Schema, "{{.DatabaseName}}", "`_vt_preflight`", -1)
initialCopySQL += s + ";\n"
}
}

vitess/go/vt/mysqlctl/tmutils/schema.go

Lines 213 to 234 in b65cde1

// Backtick database name since keyspace names appear in the routing rules, and they might need to be escaped.
// We unescape() them first in case we have an explicitly escaped string was specified.
createDatabaseSQL := strings.Replace(sd.DatabaseSchema, "`{{.DatabaseName}}`", "{{.DatabaseName}}", -1)
createDatabaseSQL = strings.Replace(createDatabaseSQL, "{{.DatabaseName}}", sqlescape.EscapeID("{{.DatabaseName}}"), -1)
sqlStrings = append(sqlStrings, createDatabaseSQL)
for _, td := range sd.TableDefinitions {
if schema.IsInternalOperationTableName(td.Name) {
continue
}
if td.Type == TableView {
createViewSQL = append(createViewSQL, td.Schema)
} else {
lines := strings.Split(td.Schema, "\n")
for i, line := range lines {
if strings.HasPrefix(line, "CREATE TABLE `") {
lines[i] = strings.Replace(line, "CREATE TABLE `", "CREATE TABLE `{{.DatabaseName}}`.`", 1)
}
}
sqlStrings = append(sqlStrings, strings.Join(lines, "\n"))
}
}

vitess/go/vt/wrangler/schema.go

Lines 283 to 315 in b65cde1

// applySQLShard applies a given SQL change on a given tablet alias. It allows executing arbitrary
// SQL statements, but doesn't return any results, so it's only useful for SQL statements
// that would be run for their effects (e.g., CREATE).
// It works by applying the SQL statement on the shard's primary tablet with replication turned on.
// Thus it should be used only for changes that can be applied on a live instance without causing issues;
// it shouldn't be used for anything that will require a pivot.
// The SQL statement string is expected to have {{.DatabaseName}} in place of the actual db name.
func (wr *Wrangler) applySQLShard(ctx context.Context, tabletInfo *topo.TabletInfo, change string) error {
filledChange, err := fillStringTemplate(change, map[string]string{"DatabaseName": tabletInfo.DbName()})
if err != nil {
return fmt.Errorf("fillStringTemplate failed: %v", err)
}
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
// Need to make sure that replication is enabled since we're only applying the statement on primaries
_, err = wr.tmc.ApplySchema(ctx, tabletInfo.Tablet, &tmutils.SchemaChange{
SQL: filledChange,
Force: false,
AllowReplication: true,
SQLMode: vreplication.SQLMode,
})
return err
}
// fillStringTemplate returns the string template filled
func fillStringTemplate(tmpl string, vars any) (string, error) {
myTemplate := template.Must(template.New("").Parse(tmpl))
data := new(bytes.Buffer)
if err := myTemplate.Execute(data, vars); err != nil {
return "", err
}
return data.String(), nil
}

@shlomi-noach
use 'sqlparser.ReplaceTableQualifiers()' to inject placeholder, then …
2e93538 
…'UnqualifyDatabaseNamePlaceholder()'. A bit of all-round refactoring.

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
allow changing the empty qualifier
f13fc87 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
more test cases
4e2f9a0 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
fix unit tests (typo in table3 creation), adapt to allowing empty sch…
76e5cf3 
…ema to translate into qualified schema

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
PreflightSchemaChange is deprecated
62921bd 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
check error value
2390702 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Use CanonicalString
0ac3f49 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach removed NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says NeedsIssue A linked issue is missing for this Pull Request labels Aug 29, 2023
@shlomi-noach
Copy link
Contributor Author

@dbussink I don't think

vitess/go/vt/wrangler/schema.go

Lines 283 to 315 in b65cde1

// applySQLShard applies a given SQL change on a given tablet alias. It allows executing arbitrary
// SQL statements, but doesn't return any results, so it's only useful for SQL statements
// that would be run for their effects (e.g., CREATE).
// It works by applying the SQL statement on the shard's primary tablet with replication turned on.
// Thus it should be used only for changes that can be applied on a live instance without causing issues;
// it shouldn't be used for anything that will require a pivot.
// The SQL statement string is expected to have {{.DatabaseName}} in place of the actual db name.
func (wr *Wrangler) applySQLShard(ctx context.Context, tabletInfo *topo.TabletInfo, change string) error {
filledChange, err := fillStringTemplate(change, map[string]string{"DatabaseName": tabletInfo.DbName()})
if err != nil {
return fmt.Errorf("fillStringTemplate failed: %v", err)
}
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
// Need to make sure that replication is enabled since we're only applying the statement on primaries
_, err = wr.tmc.ApplySchema(ctx, tabletInfo.Tablet, &tmutils.SchemaChange{
SQL: filledChange,
Force: false,
AllowReplication: true,
SQLMode: vreplication.SQLMode,
})
return err
}
// fillStringTemplate returns the string template filled
func fillStringTemplate(tmpl string, vars any) (string, error) {
myTemplate := template.Must(template.New("").Parse(tmpl))
data := new(bytes.Buffer)
if err := myTemplate.Execute(data, vars); err != nil {
return "", err
}
return data.String(), nil
}
is related; looks like an altogether different mechanism (using text/template, haven't seen this elsewhere in vitess code), and not part of {{.DatabaseName}} thing.

@shlomi-noach
normalize expected queries
7d30621 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
normalize definition
d170ce9 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
normalize definition
bad39ff 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Copy link
Contributor Author

@dbussink After an hour of debugging a test failure, I came full circle to recognize that

vitess/go/vt/wrangler/schema.go

Lines 283 to 315 in b65cde1

// applySQLShard applies a given SQL change on a given tablet alias. It allows executing arbitrary
// SQL statements, but doesn't return any results, so it's only useful for SQL statements
// that would be run for their effects (e.g., CREATE).
// It works by applying the SQL statement on the shard's primary tablet with replication turned on.
// Thus it should be used only for changes that can be applied on a live instance without causing issues;
// it shouldn't be used for anything that will require a pivot.
// The SQL statement string is expected to have {{.DatabaseName}} in place of the actual db name.
func (wr *Wrangler) applySQLShard(ctx context.Context, tabletInfo *topo.TabletInfo, change string) error {
filledChange, err := fillStringTemplate(change, map[string]string{"DatabaseName": tabletInfo.DbName()})
if err != nil {
return fmt.Errorf("fillStringTemplate failed: %v", err)
}
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
// Need to make sure that replication is enabled since we're only applying the statement on primaries
_, err = wr.tmc.ApplySchema(ctx, tabletInfo.Tablet, &tmutils.SchemaChange{
SQL: filledChange,
Force: false,
AllowReplication: true,
SQLMode: vreplication.SQLMode,
})
return err
}
// fillStringTemplate returns the string template filled
func fillStringTemplate(tmpl string, vars any) (string, error) {
myTemplate := template.Must(template.New("").Parse(tmpl))
data := new(bytes.Buffer)
if err := myTemplate.Execute(data, vars); err != nil {
return "", err
}
return data.String(), nil
}
is very much related 🤕

@dbussink
Copy link
Contributor

@dbussink After an hour of debugging a test failure, I came full circle to recognize that

Ah yes, the syntax of {{.DatabaseName}} comes from it being used as a Go template variable in the first place.

@shlomi-noach
handle CreateDatabase, AlterDatabase, DropDatabase statements
cb8100a 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
slight refactor
6db7847 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Copy link
Contributor Author

OK, good to go! Looking for reviews. Formatting is back to String() and all tests adapted.

deepthi
deepthi approved these changes Sep 19, 2023
View reviewed changes
Copy link
Member

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM except for the one question about a comment.

go/vt/sqlparser/utils.go Outdated
@@ -118,6 +118,7 @@ func NormalizeAlphabetically(query string) (normalized string, err error) {
// replaces any cases of the provided database name with the
// specified replacement name.
// Note: both database names provided should be unescaped strings.
// The returned query is formatted canonically (will look different than original query)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this comment still valid? Since you reverted the call to CanonicalString?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@mattlord
Test auto_increment clause removal as well
0be16bf 
Signed-off-by: Matt Lord <mattalord@gmail.com>
mattlord
mattlord approved these changes Sep 19, 2023
View reviewed changes
Copy link
Contributor

@mattlord mattlord left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I only had a minor comment and will let you resolve that as you feel best.

go/vt/mysqlctl/schema.go Outdated
@@ -249,6 +250,27 @@ func (mysqld *Mysqld) collectSchema(ctx context.Context, dbName, tableName, tabl
return fields, columns, schema, nil
}

// normalizedStatement returns a table schema with database names replaced, and auto_increment annotations removed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment doesn't match the behavior, does it? We only do that replacement for views AFAICT. Which one is correct, the comment or the code? I think it's the comment as the code was just moved.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

go/vt/sqlparser/utils.go
Comment on lines +144 to +161
case *CreateDatabase:
if node.DBName.String() == oldQualifier.String() {
node.DBName = newQualifier
cursor.Replace(node)
modified = true
}
case *AlterDatabase:
if node.DBName.String() == oldQualifier.String() {
node.DBName = newQualifier
cursor.Replace(node)
modified = true
}
case *DropDatabase:
if node.DBName.String() == oldQualifier.String() {
node.DBName = newQualifier
cursor.Replace(node)
modified = true
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason not to consolidate these?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - golang will give compilation errors, because SQLNode (the shared interface for all these types) does not have a DBName field (obviously as it is an interface, and each implementing type adds the fields independently).

go/vt/sqlparser/utils_test.go
in: "CREATE TABLE t (id int primary key)",
out: "CREATE TABLE t (id int primary key)",
},
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is escaped but not qualified.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I follow?

@mattlord
Merge remote-tracking branch 'origin/main' into safe-table-name-repla…
ce04269 
…cement

Signed-off-by: Matt Lord <mattalord@gmail.com>
@mattlord
Adjust usage of function added after PR branch
8fbeb88 
Signed-off-by: Matt Lord <mattalord@gmail.com>
@mattlord mattlord requested a review from ajm188 as a code owner September 19, 2023 22:26
@shlomi-noach
updated comment
b3b3882 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
updated comment
4b65366 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach marked this pull request as draft September 21, 2023 05:13
@shlomi-noach
Copy link
Contributor Author

Temporarily converted to draft because I want to confirm something before merging, and I don't want anyone else to accidentally merge in the interim. Will likely merge this shortly.

@frouioui frouioui modified the milestones: v18.0.0, v19.0.0 Sep 29, 2023
@github-actions
Copy link
Contributor

This PR is being marked as stale because it has been open for 30 days with no activity. To rectify, you may do any of the following:

  • Push additional commits to the associated branch.
  • Remove the stale label.
  • Add a comment indicating why it is not stale.

If no action is taken within 7 days, this PR will be closed.

@github-actions github-actions bot added the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Oct 30, 2023
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2023

This PR was closed because it has been stale for 7 days with no activity.

@github-actions github-actions bot closed this Nov 6, 2023
@shlomi-noach shlomi-noach removed the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Nov 7, 2023
@shlomi-noach shlomi-noach reopened this Nov 7, 2023
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 8, 2023

This PR is being marked as stale because it has been open for 30 days with no activity. To rectify, you may do any of the following:

  • Push additional commits to the associated branch.
  • Remove the stale label.
  • Add a comment indicating why it is not stale.

If no action is taken within 7 days, this PR will be closed.

@github-actions github-actions bot added the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Dec 8, 2023
@shlomi-noach shlomi-noach removed the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Dec 11, 2023
@github-actions GitHub Actions
Copy link
Contributor

This PR is being marked as stale because it has been open for 30 days with no activity. To rectify, you may do any of the following:

  • Push additional commits to the associated branch.
  • Remove the stale label.
  • Add a comment indicating why it is not stale.

If no action is taken within 7 days, this PR will be closed.

@github-actions github-actions bot added the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Jan 11, 2024
@github-actions GitHub Actions
Copy link
Contributor

This PR was closed because it has been stale for 7 days with no activity.

@github-actions github-actions bot closed this Jan 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mattlord mattlord mattlord approved these changes

@deepthi deepthi deepthi approved these changes

@dbussink dbussink Awaiting requested review from dbussink

@rohit-nayak-ps rohit-nayak-ps Awaiting requested review from rohit-nayak-ps rohit-nayak-ps is a code owner

@harshit-gangal harshit-gangal Awaiting requested review from harshit-gangal harshit-gangal is a code owner

@systay systay Awaiting requested review from systay systay is a code owner

@GuptaManan100 GuptaManan100 Awaiting requested review from GuptaManan100 GuptaManan100 is a code owner

@ajm188 ajm188 Awaiting requested review from ajm188

Assignees
No one assigned
Labels
Component: General Changes throughout the code base Component: Query Serving Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. Type: Bug
Projects
None yet
Milestone
v19.0.0
Development

Successfully merging this pull request may close these issues.

Bug Report: Mysqld.normalizedSchema() erroneously replaces table name with placeholder if same as DB name
5 participants
@shlomi-noach @dbussink @deepthi @mattlord @frouioui